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ABSTRACT 

As the third-party planner and implementer of higher education 
institutional and program evaluations, the Higher Education 
Evaluation and Accreditation Council of Taiwan (HEEACT) 
completed program evaluations for all 145 undergraduate business 
and management (B&M) programs in 43 universities/colleges from 
2006 to 2010. In the 145 programs evaluated, 105 programs (or 72 
percent) received a five-year certificate of accreditation. However, 
the failure rate (28 percent) of the programs in this field is much 
higher than the average failure rate of overall university programs 
(13 percent). This indicates that not only all programs in this field 
should keep enhancing quality assurance but also around one third of 
programs among them have to make many efforts to meet the 
accreditation criteria. The purpose of this paper was to identify the 
critical factors of passing/failing the program evaluation and the 
measures in line with the critical factors should be urgently 
improved. To attain the purpose, a content analysis of 145 B&M 
program evaluation reports was conducted. As result of the analysis, 
the following results are achieved: (1) The 10 critical factors of 
passing/failing the program evaluation are coherence between goal 
feature and curriculum, self-improvement capability and mechanism, 
composition and operation of program committee, curriculum design 
and mapping, interaction between faculty and student, use and 
benefit of instructional equipment, environment and rewarding 
system for research, research outcome and faculty load, graduate’s 
education-employment match, and employment counseling and 
alumni follow-up; (2) The three measures that should be urgently 
taken include “aligning program goals, faculty quantity and quality, 
and curriculum planning”, “encouraging all faculty to engage in 
research and increase research quantity, quality and 
internationalization”, as well as “strengthening faculty quantity and 
quality, and lightening their workload”. 
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Introduction 

Program evaluation has become an essential element of the means of continuous improvement 
and accountability measures of universities. In Taiwan, there are the following two tracks for 
post-compulsory education: academic education and technological and vocational education. 
The official program evaluation for the 78 universities/colleges in the track of academic 
education is conducted by the Higher Education Evaluation and Accreditation Council of 
Taiwan (HEEACT). It is accreditation-oriented, adopting the PDCA (Plan-Do-Check- Act) 
model with an emphasis on program self-positioning, student learning outcome, and 
mechanism for continuous improvement. 

The spirit of the program evaluation was “to ensure that the program provides students 
with a quality learning environment." Therefore, the design of program evaluation standards 
mainly focused on the following four aspects: (1) What to do — What are the aims and goals of 
the program? (2) How to do — How does the program enhance its curriculum design, faculty 
quality and teaching, student learning and assessment, as well as administration and 
management? (3) What are achieved — To what extent do faculty professional performance and 
graduates’ performance meet program aims and goals and what are the program features? (4) 
How to improve — How does the program make improvement when any operation of the 
program deviates from program aims and goals and what is the effectiveness of improvement? 
That is to say, the program evaluation standards were centered on students’ “right to education” 
(HEEACT, 2009). HEEACT’ s review procedure of program evaluation is shown as Figure 1. 
Obviously, there are the following three tiers of decision making teams in charge of the result 
of any program evaluation: program site-visit team, discipline-specific accreditation taskforce, 
and accreditation commission. 
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Figure 1 . The review procedure of program evaluation/accreditation 
Source: Lee, Wei, & Wang (2013), p. 8. 


The first round of university program evaluation was conducted from 2006 to 2010. In 
total, more than 4,000 programs (including 2,822 undergraduate programs) in 78 
universities/colleges evaluated in the first round were categorized into 44 disciplines. In the 78 
universities/colleges during the first-round evaluation, 4 3 Tad a (or several) business and 
management (B&M) program(s)— bachelor, master, doctoral or cross-departmental program(s). 
Totally, there were 286 B&M programs in which accounted for 145 undergraduate programs. 
There were three possible results of program evaluation: Pass, Referred or Fail. In Taiwanese 
culture, “Referred” is often considered as “Fail” because it also fails to pass. 

There were 2,822 undergraduate programs evaluated in the first round of program 
evaluation. Among them, 2,445 (87 percent) passed the evaluation and received a five-year 
certificate of accreditation, while 377 (13 percent) failed to pass (see Figure 2). 
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B&M is one of the 44 disciplines categorized in the first-round program evaluation and 
had the following four sub-areas: finance, accounting, general management (including 
organizational and human resource management, strategic and international business 
management, and technology management) and medical management. In the 145 
undergraduate B&M programs evaluated, 105 programs (72 percent) received a five-year 
certificate of accreditation. However, the failure rate (28 percent) of the programs in this 
discipline/field is much higher than the average failure rate of overall university programs (13 
percent; see Figure 2). It implies that about one-third of undergraduate B&M programs did not 
provide their students with a quality learning environment. The purpose of this paper was to 
identify the critical factors of passing/failing the program evaluation and the measures in line 
with the critical factors that should be urgently taken. 

Methodology 

Content analysis was employed to attain the purpose of this study. Originally used in the field 
of communication research, content analysis is an objective, systematic and qualitative method 
based on a quantitative description of explicit contents in written texts. A content analysis of 
145 B&M program evaluation reports, posted on HEEACT’s website, was conducted in this 
study in 2014. 

The six main steps of content analysis utilized in this study are as follows: stating 
research question(s), selecting the text material, reading and reviewing the material selected, 
defining the unit of analysis and categories, coding the textual material, marking the keywords 
or phrases with a computer software and placing them in the categories which have been 
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identified, and interpreting and reporting the findings (Hall, u.d.; Mayring, 2000). HEEACT’s 
program evaluation reports are presented in qualitative description, which mainly include 
evaluation result and a list of the programs’ merits and suggestions for improve. In terms of 
categories, the following five aspects of program evaluation were set by HEEACT: (1) Goal, 
features and self-improvement, (2) Curriculum design and faculty teaching, (3) Student 
learning and student affairs, (4) Research and professional performance, and (5) Graduates’ 
performance. Thus, they were selected as the main topics of categories in this study. 
Furthermore, the following seven evaluation criteria were set by HEEACT: (1) Program aims 
and goals are specific and meet the professional trends of the discipline which the program 
belongs to; (2) Curriculum planning of the program reflects educational goals of the program; 
(3) Faculty quality and quantity meet the needs to attain educational goals and implement the 
curriculum planned; (4) Learning resources meet the needs for faculty teaching and student 
learning; (5) Both quality and quantity of faculty research and professional performance are 
good; (6) The mechanism of program self-improvement is sound and workable; and (7) The 
mechanism to follow-up graduates’ careers is good. Hence, the seven criteria were selected as 
the sub-topics of categories in this study. In addition, the intercoder reliability from a pilot test 
of this study was 0.874. It is acceptable according to Kassarjian (1977). 

Results 

Based on the open coding (n = 5,568) and selective coding (n = 231) on the 145 undergraduate 
B&M program evaluation reports, the results of this study were inducted as follows: 

The critical factors of passing/failing the program evaluation 

Based on the frequency-of-mention in the 145 program evaluation reports, the 10 critical 
factors of passing/failing the program evaluation are identified as follows: coherence between 
goal feature and curriculum, self-improvement capability and mechanism, composition and 
operation of program committee, curriculum design and mapping, interaction between faculty 
and student, use and benefit of instructional equipment, environment and rewarding system for 
research, research outcome and faculty load, graduate’s education-employment match, and 
employment counseling and alumni follow-up (see Table 1). With regard to the factors 
mentioned in the programs that passed the evaluation, the top three factors in descending order 
of frequency-of-mention are “Coherence between goal feature and curriculum”, “Curriculum 
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design and mapping” and “Interaction between faculty and student”. In regard to the factors 
mentioned in the programs that failed the evaluation, the top three factors in descending order 
of frequency-of-mention are "Research outcome and faculty load", “Curriculum design and 
mapping” and “Coherence between goal feature and curriculum”. Apparently, the two factors, 
“Coherence between goal feature and curriculum” and “Curriculum design and mapping”, 
are highly concerned for either the programs passing the evaluation or the programs failing the 
evaluation. Obviously, curriculum is the common keyword of the two factors. 


Table 1 . The frequency-of-mention of 10 critical factors mentioned as merits or should-be-improved suggestions 



Critical Factors 

Is this factor 
mentioned 
in the 
evaluation 
reports? 
Yes/No 

Is this factor mentioned 
in the programs passing 
the evaluation? 
(N=105) 

Yes/No (%) 

Is this factor mentioned in 
the programs failing the 
evaluation? 

(N=40) 

Yes/No (%) 

1 . 

Coherence between goal feature 
and curriculum 

124/21 

89/16(84.8/15.2) 

35/5(87.2/12.5) 

2. 

Self-improvement capability and 
mechanism 

64/81 

46/59(43.8/56.2) 

18/22(45/55) 

3. 

Composition and operation of 
program committee 

85/60 

59/44(56.2/41.9) 

26/14(65/40) 

4. 

Curriculum design and mapping 

117/28 

82/23(78.1/21.9) 

35/5(87.5/12.5) 

5. 

Interaction between faculty and 
student 

107/38 

76/32(72.4/30.5) 

31/6(77.5/15) 

6. 

Use and benefit of instructional 
equipment 

96/49 

70/35(66.7/33.3) 

26/14(65/35) 

7. 

Environment and rewarding 
system for research 

94/51 

70/35(66.7/33.3) 

24/16(60/40) 

8. 

Research outcome and faculty 
load 

101/44 

66/39(62.9/37.1) 

35/5(87.5/12.5) 

9. 

Graduate’s education- 
employment match 

74/71 

60/45(57.1/42.9) 

14/26(35/65) 

10 

Employment counseling and 

alumni follow-up 

Total 

96/49 

75/33(71.4/31.4) 

105 

21/16(19/40) 

40 


Note: Evaluation outcome “Referred” was classified into “Fail”. 
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The measures that should be urgently taken 

The frequency-of-mention of 10 critical factors mentioned in the should-be-improved 
suggestions is shown as Table 2. In terms of continuous improvement, all the suggestions 
should be valued. However, the six critical factors with frequency-of-mention percentages 
higher than 50 percent in the passing or failing program reports can be identified as follows: 
“Coherence between goal feature and curriculum”, “Composition and operation of program 
committee”, “Curriculum design and mapping”, “Environment and rewarding system for 
research”, “Research outcome and faculty load” and “Employment counseling and alumni 
follow-up”. Therefore, the three measures, in line with the six critical factors, that should be 
urgently taken include “aligning program goals, faculty quantity and quality, and curriculum 
planning”, “encouraging all faculty to engage in research and increase research quantity, 
quality and internationalization”, as well as “strengthening faculty quantity and quality and 
lightening their workload”. 
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Table 2. The frequency-of-mention of 10 critical factors mentioned in the should-be-improved suggestions 



Critical Factors 

Mentioned in the 
passing program 
reports (N=105) 

Mentioned in the 
failing program 
reports (N=40) 

1. 

Coherence between goal feature and curriculum 

54 

30 



(51.4%) 

(75.0%) 

2. 

Self-improvement capability and mechanism 

17 

13 



(16.2%) 

(32.5%) 

3. 

Composition and operation of program 

37 

22 


committee 

(35.2%) 

(55.0%) 

4. 

Curriculum design and mapping 

47 

31 



(44.8%) 

(77.5%) 

5. 

Interaction between faculty and student 

11 

14 



(10.5%) 

(35.0%) 

6. 

Use and benefit of instructional equipment 

46 

18 



(43.8%) 

(28.1) 

7. 

Environment and rewarding system for research 

11 

19 



(10.5%) 

(63.3) 

8. 

Research outcome and faculty load 

63 

35 



(60.0%) 

(87.5%) 

9. 

Graduate’s education-employment match 

21 

8 



(20.0%) 

(20.0%) 

10. 

Employment counseling and alumni follow-up 

67 

24 



(63.8%) 

(60.0%) 


Total 

105 

40 
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Figure 2. The six critical factors mentioned in the should-be-improved suggestions 
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Implications 

Based on the above results, the implications of this study may be drawn as follows: 

Implications for managerial practices 

1. The 10 critical factors of passing/failing the program evaluation can serve as the key 
success factors of program evaluation. 

Key success factors (KSFs) are the combination of important facts that is required to 
accomplish desirable goals such as passing the program evaluation (BusinessDictionary, 
2014). To effectively ensure the quality and accountability of undergraduate B&M 
programs, the 10 critical factors of passing/failing the program evaluation, such as 
coherence between goal feature and curriculum, have to be valued. 

2. The three measures should be urgently taken to avoid negative comments. 

The should-be-improved suggestions somehow mean disadvantages. Thus, the three 
measures, such as “aligning program goals, faculty quantity and quality, and curriculum 
planning”, should be urgently taken by any program to avoid negative comments from 
evaluators. 

Implications for further studies 

A content analysis of the 145 undergraduate B&M programs completed in this study results in 
more information than those are presented in this paper. The authors of this paper will present 
more results and implications in the near future. Additionally, the second round of nation-wide 
university program evaluation has been implementing. A comparison of the analysis results 
between the two rounds of evaluation is suggested to be made. 
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